% !TEX root = FDS_Validation_Guide.tex

\chapter{What is Model Validation?}

Although there are various definitions of model validation, for example the one contained in ASTM E 1355~\cite{ASTM:E1355}, most define it as the process of determining how well the mathematical model predicts the actual physical phenomena of interest. Validation typically involves (1) comparing model predictions with experimental measurements, (2) quantifying the differences in light of uncertainties in both the measurements and the model inputs, and (3) deciding if the model is appropriate for the given application. This guide only does (1) and (2). Number (3) is the responsibility of the end user. To say that FDS is ``validated'' means that the end user has quantified the model uncertainty for a given application and decided that the model is appropriate. Although the FDS developers spend a considerable amount of time comparing model predictions with experimental measurements, it is ultimately the end user who decides if the model is adequate for the job at hand. 

This Guide is merely a collection of calculation results. As FDS develops, it will expand to include new experimental measurements of newly modeled physical phenomena. With each minor release of FDS (version 5.2 to 5.3, for example), the plots and graphs are all regenerated to ensure that changes to the model have not decreased the accuracy of a previous version.

The following sections discuss key issues that must be considered when deciding whether or not FDS is appropriate for a given application. It depends on (a) the scenarios of interest, (b) the predicted quantities, and (c) the desired level of accuracy. FDS can be used to model most any fire scenario and predict almost any quantity of interest, but the prediction may not be accurate because of limitations in the description of the fire physics, and also because of limited information about the fuels, geometry, and so on.


\section{Blind, Specified, and Open Validation Experiments}

ASTM E 1355~\cite{ASTM:E1355} describes three basic types of validation calculations -- {\em Blind}, {\em Specified}, and {\em Open}.
\begin{description}
\item [Blind Calculation:] The model user is provided with a basic description of the scenario to be modeled. For this application, the problem description is not exact; the model user is responsible for developing appropriate model inputs from the problem description, including additional details of the geometry, material properties, and fire description, as appropriate. Additional details necessary to simulate the scenario with a specific model are left to the judgment of the model user. In addition to illustrating the comparability of models in actual end-use conditions, this will test the ability of those who use the model to develop appropriate input data for the models.
\item [Specified Calculation:] The model user is provided with a complete detailed description of model inputs, including geometry, material properties, and fire description. As a follow-on to the blind calculation, this test provides a more careful comparison of the underlying physics in the models with a more completely specified scenario.
\item [Open Calculation:] The model user is provided with the most complete information about the scenario, including geometry, material properties, fire description, and the results of experimental tests or benchmark model runs which were used in the evaluation of the blind or specified calculations of the scenario. Deficiencies in available input (used for the blind calculation) should become most apparent with comparison of the open and blind calculation.
\end{description}
The calculations presented in this Guide all fall into the {\em Open} category. There are several reasons for this, the first being the most practical:
\begin{itemize}
\item All of the calculations presented in this Guide are re-run with each minor release of FDS (i.e., 5.3 to 5.4). The fact that the experiments have already been performed and the results are known qualify these calculations as {\em Open}.
\item Some of the calculations described in this Guide did originally fall into the {\em Specified} category because they were first performed before the experiments were conducted. However, in almost every case, the experiment was not conducted exactly as specified, and the pre-calculated results were not particularly useful in determining the accuracy of the model.
\item None of the calculations were truly {\em Blind}, even those performed prior to the experiments. The purpose of a {\em Blind} calculation is to assess the degree to which the choice of input parameters affects the outcome. However, in such cases it is impossible to discern the uncertainty associated from the choice of input parameters from that associated with the model itself. The primary purpose of this Guide is to quantify the uncertainty of the model itself, in which case {\em Blind} calculations are of little value.
\end{itemize}


\section{How to Use this Guide}

When considering whether to use FDS for a given application, do the following:
\begin{enumerate}
\item Survey Chapter~\ref{Survey_Chapter} to learn about past efforts by others to validate the model for similar applications. Keep in mind that most of the referenced validation exercises have been performed with older versions of FDS, and you may want to obtain the experimental data and the old FDS input files and redo the simulations with the version of FDS that you plan to use.
\item Identify in Chapter~\ref{Experiments_Chapter} the experimental data sets appropriate for your application. In particular, the summary of the experiments found in Section~\ref{experiment_summary} contains a table listing various non-dimensional quantities that characterize the parameters of the experiments. For example, the equivalence ratio of a compartment fire experiment indicates the degree to which the fire was over or under-ventilated. To say that the results of a given experiment are relevant to your scenario, you need to demonstrate that its parameters ``fit'' within the parameter space outlined in Table~\ref{Test_Parameters}.
\item Search the Table of Contents to find comparisons of FDS simulations with the relevant experiments. For a given experiment, there may be numerous measurements of quantities like the gas temperature, heat flux, and so on. It is a challenge to sort out all the plots and graphs of all the different quantities and come to some general conclusion. For this reason, this Guide is organized by output quantity, not by individual experiment or fire scenario. In this way, it is possible to assess, over a range of different experiments and scenarios, the performance of the model in predicting a given quantity. Overall trends and biases become much more clear when the data is organized this way.
\item Determine the accuracy of the model for given output quantities of interest listed in Table~\ref{summary_stats}. An explanation of the accuracy metrics is given in Chapter~\ref{Error_Chapter}.
\end{enumerate}
The experimental data sets and FDS input/output files described in this Guide are all managed via the on-line project archiving system. You might want to re-run examples of interest to better understand how the calculations were designed, and how changes in the various parameters might affect the results. This is known as a {\em sensitivity study}, and it is difficult to document all the parameter variations of the calculations described in this report. Thus, it is a good idea to determine which of the input parameters are particularly important.
